Within the context of online services (e.g., web services), robust error-free code may be critical. In particular, clients of an online service may include businesses and other organizations whose operations are reliant upon the online service's uptime and provision of correct output. Accordingly, prior to deploying an online service, it is common to first submit the online service's coded implementation to a period of testing within a non-production environment (e.g., a beta testing environment) before pushing the code to production. During the testing period, developers may attempt to reduce the number of bugs within the code by applying a suite of tests (e.g., regression tests, unit tests) against the code to identity bugs. Once the code is able to pass all the tests without producing any errors, the online service may be launched by deploying the code to the production environment. Moreover, if the online service is continually updated with new features and other changes, this testing-before-deployment process may be repeated for each incoming version of the online service.
One shortcoming with using a non-production environment to test code is that the test suite applied against the code may be significantly limited in scope. Because developer time is often needed to create, configure, and continually update test cases or test data, testing suites may often only focus on a minor sample of all possible situations or inputs that the online service may encounter in production. As a result, even if the code is able to pass all of the tests that are applied in the controlled environment, the code is likely unable to handle many of the possible edge cases that could be encountered in the wild.
Embodiments of the present invention address these problems and other problems individually and collectively.